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METHOD AND APPARATUS FOR SUPPRESSING CO- CHANNEL 

INTERFERENCE IN A RECEIVER 

Technical Field 

The present invention pertains to the field of receivers 
5 for digital communications. More particularly, the present 

invention pertains to canceling interference in such 
receivers, including for example digital receivers in cellular 
telephones . 

Background Art 

10 It is well known that one of the major limitations in 

cellular and other wireless telephone networks is so-called 
co-channel interference. In the case of time division multiple 
access (TDMA) networks, such as GSM (Global System for Mobile 
Communications) /GPRS (General Packet Radio Service) /EDGE 

15 (Enhanced Data Rates For GSM Evolution) , co-channel 

interference is caused mainly by what is termed frequency 
reuse, i.e. the same frequencies in the allocated spectrum are 
used in different cells, and so, depending on the reuse 
factor, a signal received by a cellular phone will contain not 

2 0 only a desired forward channel for a current cell, but also 

channels originating in other cells. 

The prior art for co-channel interference suppression 
includes: "A Receiver, 11 having International Publication 
Number WO 01/93439, with priority date of 31 May 2000, by 

25 inventors Ottersten, Kristensson, and Astely; and also 

"Methods and Apparatus for canceling co- channel interference 
in a receiving system using spatio-temporal whitening," having 
U.S. Patent Application Pub. No. US 2003/0063596, with a 
priority date of Sep. 19, 2001, by inventors Huseyin Arslan 

30 and Ali S. Khayrallah. The receiver disclosed in WO 01/93439 

is based on the well-known fact that if (co-channel) 
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interference is considered to be colored noise and the noise 
is whitened, signal gain can be achieved. The novelty of WO 
01/93439 is asserted to be a filter providing efficient 
whitening by exploiting the additional degree of freedom that 
5 arises from the separation of the real and imaginary 

components of the received signal, i.e. of the in-phase and 
quadrature -phase components. The teachings of US 2003/0063596 
are similar to WO 01/93439 in respect to suppressing co- 
channel interference . 

10 In WO 01/93439 the interference is modeled as an IIR 

(infinite impulse response) process with order K, and 
(consequently) the whitening operation is performed by a 
(multidimensional) FIR (finite impulse response) filter with K 
(or K+l) filter taps. After the whitening operation, the 

15 impulse response of the wanted signal is of course modified; 

in particular, because of the convolution with the whitening 
filter, the whitening operation of WO 01/93439 suffers from 
what is here called "the increased channel length problem," 
i.e. the impulse response of the wanted signal becomes longer, 

20 requiring a more complex equalizer or at least a modified 

equalizer including some mechanism to take into account the 
increased channel length (otherwise the receiver incurs a 
performance loss) . The increased channel length problem 
requires that the equalizer of a receiver be modified if the 

25 whitening operation per WO 01/93439 is to be implemented by 

the receiver. 

Additionally the achievable performance gain obtainable 
using the whitening operation of WO 01/93439 depends on the 
model parameter K indicating the number of taps of the FIR 
30 filter. In general the higher the K, the higher the gain, but 

if K exceed a certain threshold (which depends on the 
particular interference being suppressed and so is in 
principle not a priori known) the problem of finding the FIR 
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filter coefficients becomes ill-conditioned, i.e. the FIR 
filter can not be found. 

What is therefore needed is a more robust, less complex 
method of suppressing co-channel interference based on noise 
5 whitening, one that is easier to integrate into existing 

receivers, such a GSM (Global System for Mobile 
Communications)/ EDGE (Enhanced Data Rates for GSM Evolution) 
receivers . 

Disclosure of the Invention 

10 Accordingly, in a first aspect of the invention, a method 

is provided for use by a receiver of a wireless communication 
system in receiving over a communication channel a radio 
transmission of a number of symbols each having an in-phase 
and a quadrature component, the method including a step of 

15 receiving and sampling the radio transmission so as to provide 

a succession of samples, and also a step of filtering the 
succession of samples, the method characterized in that: the 
step of filtering the succession of samples includes a step of 
whitening the samples on a sample-by- sample basis by 

20 evaluating, for each sample in the succession of samples, a 

noise plus interference correlation matrix including 
information about the correlation of both the in-phase and 
quadrature phase components of the sample. 

In accord with the first aspect of the invention, the 
25 step of filtering may be further characterized by: a step of 

switching on or off the step of whitening the samples, with 
the switching based on determining whether the communication 
channel is sensitivity-limited so that noise is present that 
can be characterized as substantially white. 

30 Also in accord with the first aspect of the invention, 

whether the communication channel is determined to be 
sensitivity-limited may be based on a calculated value of a 
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metric and a corresponding predetermined threshold. Further, 
the metric may be based on relative values of different 
components of the noise plus interference correlation matrix. 
Also further, the switching may be based on comparing the 
5 value of a metric defined by 

M .=&L 

K Oii 

where ^ 0 /i=Mm*J an< ^ = J • st iH also further, the 

switching may be based on examining second order or higher order 
statistic of the noise plus interference signal related to the 
10 noise plus interference correlation matrix. 

Also in accord with the first aspect of the invention, 
the method may be further characterized in that the noise plus 
interference correlation matrix is determined using: 



15 where is a noise plus interference signal. Further, the 

method may be further characterized in that each vector y k 
representing one symbol is whitened using: 

y t =wy t 

where W is defined as the inverse of a square root operation 
20 on the noise plus interference correlation matrix R l7 , so that: 

W = R' 1 2 . 

Also in accord with the first aspect of the invention, 
each symbol may be indicated by one or more samples, including 
samples from possibly different antennas. 

2 5 A second aspect of the invention provides a receiver used 

as part of or with a wireless communication system, 
characterized in that it comprises means for performing the 
steps of a method according to the first aspect of the 
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invention. Further, the receiver may be part of a mobile 
station. Also further, the receiver may be part of a base 
station of a radio access network of the wireless 
communication system. 

A third aspect of the invention provides a system 
comprising a mobile station and a base station used as part of 
or with a wireless communication system, each including a 
receiver, characterized in that at least one of the receivers 
comprises means for performing the steps of a method according 
to the first aspect of the invention. 

One of the problems that the present invention solves is 
that it does not cause the increased channel length problem, 
i.e. the number of channel taps before and after the IQ 
whitening filter are the same. 

Brief Description of the Drawings 

The above and other objects, features and advantages of 
the invention will become apparent from a consideration of the 
subsequent detailed description presented in connection with 
accompanying drawings, in which: 

Fig. 1 is a block diagram/ flow diagram of components of 
a receiver including a filter of a type in which the present 
invention can be used. 

Fig. 2 is a block diagram/ flow diagram showing the 
filter of Fig. 1 in more detail, and in particular showing a 
module for performing IQ whitening. 

Fig. 3 is a block diagram/ flow diagram of the IQ 
whitening filter module of Fig. 2. 

Best Mode For Carrying Out The Invention 

The invention provides a way to suppress co-channel 
interference in a receiver of a wireless communication system 
by use of a novel so-called IQ whitening operation and a novel 



way of modeling the interference. In particular, the whitening 
operation is done within one symbol. It is not possible to do 
whitening within one symbol using the procedure set out in the 
above-mentioned WO 01/93439 because doing so would mean 
5 setting the model parameter K (indicating the number of taps 

of the FIR filter) equal to zero, resulting in the FIR filter 
becoming an identity matrix, which does not provide any 
whitening . 

In addition, the inventors have observed that whitening 
10 an interfering signal (to suppress the interference) causes 

performance loss when the interference (or noise) is already 
substantially white, i.e. before the whitening operation. To 
avoid causing such a performance loss, the invention includes 
a switching on or off whitening depending on how white the 
15 interfering signal is. 

In the description below, it is assumed that a receiver 
in which the invention is implemented operates with two 
samples per symbol. This is the preferred implementation, but 
the invention is also applicable in case of any number of 

20 input samples per symbol. The number of samples per symbol can 

be increased also by means of interpolation if additional 
samples are not available directly from the A/D (analog to 
digital) converter in the front end of the receiver. Also, the 
description below assumes a single antenna receiver, and this 

25 is the situation in which the invention is most useful; 

however the invention can be easily extended to more than one 
receiver antenna, and the samples received from different 
antennas can be treated equivalently as fractional samples. 
For example, the algorithm would be the same if two samples 

3 0 per symbol are available from a single antenna or if one 

sample per symbol is available from two antennas. Also, still, 
although the invention is described in respect to binary PAM 
(Pulse Amplitude Modulation) , so that the symbols x are 
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restricted to the interval (-1,1), i.e. at^ . e (-1,1) , as is clear 

from the discussion that follows, nothing about the invention 
so restricts it and the invention has potential application in 
system in which any kind of binary modulation or multi level 
5 PAM (pulse amplitude modulation) is employed, including e.g. 

BPSK (binary phase shift keying) , MSK (minimum shift keying) . 
Of course the invention is also applicable for offset-QAM 
modulations like binary offset QAM and quaternary-offset QAM 
as they can be viewed as binary or quaternary PAM signal by 
10 applying a proper rotation every symbol. In particular, the 

invention is suitable for GMSK (Gaussian minimum shift keying) 
modulation used e.g. in GSM and Bluetooth, as it is well known 
that GMSK can be closely approximated by binary modulation. 

Referring now to Fig. 1, a receiver is shown of a type in 
15 which the present invention can be implemented, including: a 

front end 11 having one or more antennas and tuning equipment 
for receiving a signal on a selected communication channel and 
sampling the signal so as to provide a sampled received signal 
y(nT) where T stands for the time between (transmission of) 
20 successive symbols (i.e. the inverse of the symbol rate), and 

n~k + j where k is the index that runs for all the transmitted 
symbols, q is the oversampling index that runs from 0 to /, 

and / is the oversampling factor, i.e. the number of samples 
per symbol, (or the number of receiver antennas or the 

25 oversampling f actorxnumber of receiver antennas) ; a filter 12 

according to the invention for filtering the sampled received 
signal to remove co-channel interference and noise so as to 
produce a filtered sampled signal (i.e. a succession of 

filtered signal samples) ; and a detector 13 for determining 

30 symbols jc corresponding to the filtered signal samples 

(i.e. each symbol x corresponding to one or more samples of 
the received signal) . 
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What is indicated in Fig. 1 as the front end 11 
represents many different functionalities necessary for 
receiver operation, functionalities separable from those 
provided by the invention, such as e.g. means for channel 
5 estimation, means for frequency offset estimation, means for 

DC offset compensation, means for signal de-rotation (in case 
it is needed such as for example for GMSK modulation) . 
Basically, as indicated in Fig. 1, the front end 11 gives as 
output baseband samples y(nT) of the received signal and the 
10 estimated discrete channel impulse response h(sT) , where 

s — m + qll and m runs from 0 to v with 0 < m < v and v equal to 
one less than the channel impulse response length. 

Referring now to Fig. 2, the filter 12 according to the 
invention includes a module 12c for performing IQ whitening, 

15 but since, as mentioned above, it is not always advantageous 

to perform IQ whitening, in a preferred embodiment of the 
filter 12, a switch module 12a is included for switching 
between one of two filtering modules 12b-c depending on a 
comparison of predetermined thresholds with metrics calculated 

2 0 by the switch module 12a, as explained in more detail below. 

The switch module 12a switches between either a filter module 
12b that performs IQ whitening (either according to the 
invention or according to the prior art) , or a filter module 
12c that performs no whitening. 

25 First the invention with respect to the IQ filtering 

module 12c is described, and then the invention with respect 
to the switch 12a is described. 

In the description that follows, we assume over-sampling 
by an over- sampling factor of 1=2 and we write the received 
30 signal samples y k at the output of block 11 in terms of the 

channel data input x k and the sampled channel impulse response 
hm as : 
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(i) 



m=0 



j=\ m=0 



where 0<m<v and v is equal to one less than the channel 
impulse response length. The superscript j denotes the 

indexing for each signal on the selected communication 
channel, with j =0 indicating the desired signal and the other 

j values indicates co-channel interference, and with 
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and using T for the time between (transmission of) successive 
symbols. We can define a block of N f received samples as: 
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or more compactly, as: 
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.- XT y (0) , 
k+N f -\:k ~~ n 0 A *+A^— l:*-v T 



A/ 



x 0) + n 



k+N f -\:k ' 



(2) 



which includes the desired signal H 0 *£|^ 1:jfc _ v and the noise plus 



A/ 



(co-channel) interference signal ^H,*^ _ vk _ v +n k+N _ hk , For 

7=1 

convenience, we define the noise plus interference signal 

at the instant corresponding to index (sample counter) k as 
the 4x1 vector: 



L = 



Re{z[£r]} 
Im{z[fcr]} 
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Next, we define a filter operation L 1 so as to provide the 
whitened received signal samples y as follows: 



\:k ' 



(4) 



where, according to the invention, 
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(5) 



in which R l7 is the 4x4 noise plus interference correlation 



matrix given by 



(6) 



in which ] is the mathematical operation of taking the 

ensemble average. Since the interference arises from a cyclo 
stationary random process, the expectation operation can be 
replaced by a time average, i.e. 
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Thus, each 4x1 vector representing one symbol can be 
whitened as 

y*=w y , (7) 

where W is defined as the inverse of a square root operation 
on a positive definite Hermitian matrix R (7 , i.e. 



W = R: ,/2 . (7.1) 



In the preferred embodiment, the Hermitian matrix R /7 is the 
noise plus interference correlation matrix, and the whitening 
10 matrix W = R7. 1/2 is obtained in the preferred embodiment as the 

inverse of a Choleski factorization of R /7 . Alternatively the 
whitening matrix W can be obtained by any factorization of 
R„. , i.e. any factorization indicated by, 

(WW')" 1 = R l7 , (7.2) 
15 or be obtained by any factorization of R /7 , as indicated by, 

U2tV' = R l7 , 

such as for example by Singular Value Decomposition (SVD) . In 
such a case, the whitening matrix W can be obtained as 

W = VX"^ . 

2 0 According to the invention, a whitened impulse response 

H 0 is also provided by the filter 12, this time acting on the 
impulse response H 0 : 
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Tl -1 

H 0 — L H 0 = 



h< 0) h< 0) .. h< 0) 0 .. 0 
0 h<°> h<°> .. h[°» 0 .. 



0 .. 0 h< 0) h< 0) .. 



(8) 



Therefore, each 4x1 vector h m may be computed as: 



hi o) =Whi 0) (9) 

mm 



for 0<m<v. So, it is clear that the interference suppression 
method disclosed by the present invention does not cause the 
problem of increased channel length, i.e. it is clear from eq. 

(9) that h has the same dimension as h m . 

m m 



Referring now to Fig. 3, the filter module 12c (Fig. 2) 
according to the invention preferably includes: a first module 

10 31 for constructing the succession of samples y^. for k = l 9 ... 9 N f , 

with each k corresponding to one symbol; a next module 32 for 

determining the noise plus interference correlation matrix R l7 ; 
and a third module 33 for calculating the filtered, whitened 
samples y^Wy^ for k = l 9 ...,N f . 

15 Referring still to Fig. 3, in some embodiments the filter 

12 can include an additional module following the module 33 

for calculating y*=Wy A . Such a module would not decimate the 

signal in case of a fractionally spaced equalizer, but in case 
of a symbol-spaced equalizer, the module would decimate the 
2 0 signal, and would do so by filtering the signal with a filter 

equivalent to the feed-forward filter of a DFE equalizer or by 
a polyphase matched filter. 

In the preferred embodiment the detector 13 is a trellis 
detector, and the input of the trellis detector is symbol 
2 5 spaced. This symbol spaced input can be obtained by post 

processing the output of the whitening filter W by a pre- 
filter (designed for white noise) equivalent to the feed 
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forward filter of a Decision Feedback Equalizer (DFE) . By 
approximating the noise plus interference correlation matrix 

R /7 as in eq. (6) , only the correlation within one symbol is 

removed. Therefore, some residual correlation may be expected, 
5 and could be, for instance, taken into account in a subsequent 

optimization of a feed-forward filter. After such post 
processing, a Forney metric can be used in the trellis Viterbi 
equalizer. See e.g. "Unification of MLSE Receivers and 
Extensions to Time Varying Channels' 7 ; by G. Bottomley, S. 
10 Chennakesku; in IEEE Trans. Inform. Theory; vol. 46, pp 464- 

472, April 1998. 

Alternatively the equalizer could employ the Ungerboeck 
metric, also described in the above paper by Bottomley et al. 
In such a case, there is no need to employ a pre-filter, but 
15 the output of the whitening filter W can be decimated to one 

sample per symbol by employing a polyphase matched filter, 
matched to the whitening desired impulse response. 

Alternatively the equalizer can operate with more than 
one sample per symbol, and in such a case there is yet another 

20 alternative implementation. If the detector can operate with 

more than one sample per symbol, instead of using a pre- 
whitening operation, the same result can be obtained by 
modifying the metric inside the equalizer to take into account 
the noise correlation. More specifically a modified branch 

2 5 metric given by: 



f y V ( \ 
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V m=0 J 



can be used instead of the Euclidean metric, 



M k = 



(id 



V m-0 J V m=0 J 

which would be used after the pre-whitening operation 
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The invention also provides a mechanism for enabling and 
disabling IQ whitening (either performed as described above or 
using other techniques) . Such a switching mechanism is useful 
because, it turns out, IQ whitening to suppress co-channel 
interference is remarkably effective for binary modulated co- 
channel interference-limited channels, but causes some 
performance loss in sensitivity (white noise) limited 
channels. The invention thus provides a switching algorithm 
that can be used to dynamically switch between IQ whitening 
and non-whitening in a receiver. In some embodiments, the 
switching is based on examining relative values of different 
components of the noise plus interference correlation matrix 

R„ , as explained below. In some other embodiments, the 
switching is based on examining expectations /?[...] of different 

products and i k i fc+1 , as also explained below. The idea 

behind the switching mechanism provided by the invention is to 
determine whether the noise present is white noise (in such 
case the whitening is turned off), or non white noise, i.e. 
whether it is temporally correlated (in such case the 
whitening is turned on) . 

In the embodiment based on examining relative values of 
different components of the noise plus interference 

correlation matrix R l7 , we detect whether the noise is white as 

follows. As described above, the IQ whitening filter (with 
truncated autocorrelation) requires the computation of the 4x4 

(2-IQ split, 2 -over -sampling) correlation matrix R„ from the 

noise samples. So in the preferred embodiment, the metric for 
switching on and off whitening is derived as a function of the 

elements of R lY . 

In the case of binary modulated co-channel interferers, 
it turns out that the temporal correlation (between the 
fractional samples, i.e. the samples within one symbol) can be 
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expressed as a combination of the elements of the IQ 
correlation matrix K n , as follows: 

Rtemporal=Rii (1,3)+R B (2 , 4 ) + V^T * ( R.. (2 , 3 ) - R.. ( 1 , 4 ) ) . (12) 

Thus for a binary modulated interfering signal, the temporal 
correlation can be obtained from the elements of the IQ 
correlation matrix, and the correlation between the real and 
imaginary components of a sample does not become zero. 

The switching provided by the invention according to the 
embodiment based on examining relative values of different 
components of the noise plus interference correlation matrix 

R J( can thus be prescribed as follows. Define 



M tc =2 



(Rfi 0,3) + R„ (2,4)) + V^T * (R fi (2,3) - Rj,- (1,4)) 



J] (R a (2*,2*) + R.. (2k - 1, 2k- 1)) 



(13) 



as a metric for indicating whether a channel is temporally 
correlated (i.e. is sensitivity limited so that the noise is 
white noise) . As can be seen, the metric depends on the 
relative values of different components of the noise plus 

interference correlation matrix R i7 . Then, if M tc >r tc for some 

predetermined threshold r tc , the channel is categorized as 

temporally correlated and whitening is performed; otherwise 
whitening is disabled. 

As mentioned, in some other embodiments the switching is 
based on examining expectations /?[...] of different products i^i^ 

and \ k \ M . In such embodiments, we evaluate the (numerical) 
value, 



Rou=4lh\ (14) 



and also the (numerical) value, 



Rui =E[i* k i k+l \. (15) 
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Then, using as a metric for determining whether to whiten the 
quantity M ti defined by: 

M ri =A, (16) 

we switch to whitening when M ti is greater than some 
5 predetermined threshold r ti (having a value that can depend on 

the application) . 

Both of the embodiments for switching described above are 
based on observation of the second order statistics of the 
interference signal. Other embodiments may depend a more 
10 complicated measure. 

Thus, referring again to Fig. 2, in case of insubstantial 
temporal correlation (i.e. e.g. in case of M tc <r tc ), the switch 

12a switches to the module 12c for filtering without 
whitening, and otherwise switches to the module 12b for 
15 whitening. 

It is to be understood that the above-described 
arrangements are only illustrative of the application of the 
principles of the present invention. Numerous modifications 
and alternative arrangements may be devised by those skilled 
2 0 in the art without departing from the scope of the present 

invention, and the appended claims are intended to cover such 
modifications and arrangements. 
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